Methods of and apparatuses for recognizing motion of objects, and associated systems

ABSTRACT

A method of recognizing motion of an object may include periodically obtaining depth data of a first resolution and two-dimensional data of a second resolution with respect to a scene using an image capturing device, wherein the second resolution is higher than the first resolution; determining a motion tracking region by recognizing a target object in the scene based on the depth data, such that the motion tracking region corresponds to a portion of a frame and the portion includes the target object; periodically obtaining tracking region data of the second resolution corresponding to the motion tracking region; and/or analyzing the motion of the target object based on the tracking region data.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. application Ser. No.14/064,639, filed on Oct. 28, 2013, which claims priority from KoreanPatent Application No. 10-2012-0120883, filed on Oct. 30, 2012, in theKorean Intellectual Property Office (KIPO), the entire contents of eachof which are incorporated herein by reference.

BACKGROUND 1. Field

Some example embodiments may relate generally to processing of imagedata. Some example embodiments may relate to methods of and/orapparatuses for recognizing motion of objects based on depth data and/ortwo-dimensional data.

2. Description of Related Art

A two-dimensional image sensor may be used to obtain two-dimensionaldata and the two-dimensional data may be used to recognize a shapeand/or a motion of an object. Particularly the technology forrecognizing the motion of a user is developed to support a userinterface. The two-dimensional data for the motion recognition mayinclude color image data or black and white image data.

Alternatively a depth sensor may be used to obtain depth data and thedepth data may be used to recognize the shape and/or the motion of theobject. The depth data for the motion recognition may includeinformation of a distance to the object from the sensor.

In general, the two-dimensional data may be provided with relatively ahigher resolution, but it is difficult to distinguish the object fromthe background based on the two-dimensional data during the data processfor the motion recognition. The depth data may be provided withrelatively a lower resolution and thus it is difficult to discern thecomplex shape of the object based on the depth data during the dataprocess for the motion recognition.

SUMMARY

Some example embodiments of the inventive concept may provide methods ofrecognizing motion of objects, capable of discerning the motion of theobjects efficiently based on depth data and/or two-dimensional data.

Some example embodiments of the inventive concept may provideapparatuses for recognizing motion of objects, capable of discerning themotion of the objects efficiently based on depth data and/ortwo-dimensional data.

Some example embodiments of the inventive concept may provide systemsadopting the methods and/or apparatuses of recognizing the motion of theobjects.

In some example embodiments, a method of recognizing motion of an objectmay comprise periodically obtaining depth data of a first resolution andtwo-dimensional data of a second resolution with respect to a sceneusing an image capturing device, wherein the second resolution is higherthan the first resolution; determining a motion tracking region byrecognizing a target object in the scene based on the depth data, suchthat the motion tracking region corresponds to a portion of a frame andthe portion includes the target object; periodically obtaining trackingregion data of the second resolution corresponding to the motiontracking region; and/or analyzing the motion of the target object basedon the tracking region data.

In some example embodiments, periodically obtaining the depth data andthe two-dimensional data may comprise providing the depth datacorresponding to the frame with a first frame period using depth pixelsof the first resolution; and/or providing the two-dimensional datacorresponding to the frame with a second frame period using color pixelsof the second resolution.

In some example embodiments, the method may further comprisesynchronizing the depth data and the two-dimensional data to be matchedwith each other, when the first frame period is different from thesecond frame period.

In some example embodiments, the tracking region data corresponding tothe motion tracking region may be provided with the first frame periodor the second frame period.

In some example embodiments, periodically obtaining the tracking regiondata of the second resolution may comprise extracting region image dataof the second resolution corresponding to the motion tracking regionfrom the two-dimensional data of the second resolution corresponding tothe frame; and/or providing the region image data of the secondresolution as the tracking region data.

In some example embodiments, periodically obtaining the tracking regiondata of the second resolution may comprise extracting region depth dataof the first resolution corresponding to the motion tracking region fromthe depth data of the first resolution corresponding to the frame;extracting region image data of the second resolution corresponding tothe motion tracking region from the two-dimensional data of the secondresolution corresponding to the frame; compensating for the region depthdata of the first resolution using the region image data of the secondresolution to generate region depth data of the second resolution;and/or providing the region depth data of the second resolution as thetracking region data.

In some example embodiments, the depth pixels and the color pixels maybe arranged in a common pixel array.

In some example embodiments, the depth pixels and the color pixels maybe arranged respectively in distinct pixel arrays that are spaced apartfrom each other.

In some example embodiments, periodically obtaining the depth data andthe two-dimensional data may comprise periodically providing raw datacorresponding to the frame using time-of-flight (TOF) depth pixels, theTOF depth pixels operating in response to a plurality of demodulationsignals having different phases from each other; and/or calculating thedepth data of the first resolution and the two-dimensional data of thesecond resolution based on the raw data.

In some example embodiments, calculating the depth data of the firstresolution and the two-dimensional data of the second resolution maycomprise providing the depth data of the first resolution by combiningevery M bits of the raw data, where M is a positive integer equal to orgreater than two and/or providing the two-dimensional data of the secondresolution by combining every N bits of the raw data, where N is apositive integer equal to or smaller than M.

In some example embodiments, the demodulation signals may have phasedifference of 0, 90, 180 and 270 degrees, respectively, with respect totransmission light radiated from the image capturing device.

In some example embodiments, providing the depth data of the firstresolution may comprise providing one bit value of the depth data basedon four bit values of the raw data, the four bit values respectivelycorresponding to the four demodulation signals having the phasedifference of 0, 90, 180 and 270 degrees, respectively.

In some example embodiments, providing the two-dimensional data of thesecond resolution may comprise providing one bit value of thetwo-dimensional data by summing two bit values of the raw data, the twobit values respectively corresponding to the two demodulation signalshaving the phase differences of 0 and 180 degrees; and/or providinganother bit value of the two-dimensional data by summing other two bitvalues of the raw data, the other two bit values respectivelycorresponding to the two demodulation signals having the phasedifferences of 90 and 270 degrees.

In some example embodiments, periodically obtaining the tracking regiondata of the second resolution may comprise extracting region depth dataof the first resolution corresponding to the motion tracking region fromthe depth data of the first resolution corresponding to the frame;extracting region image data of the second resolution corresponding tothe motion tracking region from the two-dimensional data of the secondresolution corresponding to the frame; compensating for the region depthdata of the first resolution using the region image data of the secondresolution to generate region depth data of the second resolution;and/or providing the region depth data of the second resolution as thetracking region data.

In some example embodiments, determining the motion tracking region maycomprise determining coordinates of a center point of the motiontracking region in the frame; and/or determining a size of the motiontracking region in the frame.

In some example embodiments, the method may further comprise upgradingthe motion tracking region according to the motion of the target object.

In some example embodiments, upgrading the motion tracking region maycomprise detecting a change of a position of the target object in thescene based on the depth data and/or changing coordinates of a centerpoint of the motion tracking region in the frame based on the change ofthe position of the target object in the scene.

In some example embodiments, upgrading the motion tracking region maycomprise detecting a change of distance to the target object based onthe depth data; decreasing a size of the motion tracking region when thedistance to the target object increases; and/or increasing the size ofthe motion tracking region when the distance to the target objectdecreases.

In some example embodiments, an apparatus for recognizing motion of anobject may comprise an image capturing device configured to periodicallyprovide depth data of a first resolution and two-dimensional data of asecond resolution with respect to a scene, wherein the second resolutionis higher than the first resolution; a motion region tracker configuredto determine a motion tracking region by recognizing a target object inthe scene based on the depth data, such that the motion tracking regioncorresponds to a portion of a frame and the portion includes the targetobject, and configured to periodically provide tracking region data ofthe second resolution corresponding to the motion tracking region;and/or a motion analyzer configured to analyze the motion of the targetobject based on the tracking region data.

In some example embodiments, the image capturing device may comprise apixel array in which depth pixels of the first resolution and colorpixels of the second resolution are alternatively arranged, the depthpixels providing the depth data with a first frame period, and the colorpixels providing the two-dimensional data with a second frame period.

In some example embodiments, the image capturing device may comprise afirst pixel array in which depth pixels of the first resolution arearranged, the depth pixels providing the depth data with a first frameperiod; and/or a second pixel array in which color pixels of the secondresolution are arranged, the color pixels providing the two-dimensionaldata with a second frame period.

In some example embodiments, the image capturing device may comprise apixel array in which time-of-flight (TOF) depth pixels are arranged, theTOF depth pixels operating in response to a plurality of demodulationsignals having different phases from each other to periodically provideraw data corresponding to the frame.

In some example embodiments, the demodulation signals may have phasedifference of 0, 90, 180, and 270 degrees, respectively, with respect totransmission light radiated from the image capturing device, and/or onebit value of the depth data may be provided based on four bit values ofthe raw data, the four bit values respectively corresponding to the fourdemodulation signals having the phase difference of 0, 90, 180, and 270degrees, respectively.

In some example embodiments, a system may comprise an image capturingdevice configured to periodically provide depth data of a firstresolution corresponding to a frame of a scene and two-dimensional dataof a second resolution corresponding to the frame, wherein the secondresolution is higher than the first resolution; a motion region trackerconfigured to determine a motion tracking region by recognizing a targetobject in the scene based on the depth data, such that the motiontracking region corresponds to a portion of the frame and the portionincludes the target object, and configured to periodically providetracking region data of the second resolution corresponding to themotion tracking region; a motion analyzer configured to analyze motionof the target object based on the tracking region data; and/or a controldevice configured to generate an event corresponding to the motion ofthe target object based on an analysis result of the motion analyzer.

In some example embodiments, the system may be a user interface systemthat operates by recognizing motion of a user. The target object mayinclude a body of the user or a portion of the body of the user.

In some example embodiments, an apparatus for recognizing motion of anobject may comprise a first device configured to provide depth data fora scene that includes the object at a first resolution andtwo-dimensional data for the scene at a second resolution; a seconddevice configured to determine a motion tracking region by recognizingthe object based on the depth data and configured to provide trackingregion data of the second resolution corresponding to the motiontracking region; and/or a third device configured to analyze the motionof the object based on the tracking region data. The second resolutionmay be higher than the first resolution. The motion tracking region maycorrespond to a portion of a frame. The portion of the frame may includethe object.

In some example embodiments, the first device may comprise a sensingunit. The sensing unit may comprise a depth pixel array. The depth pixelarray may be configured to output depth information.

In some example embodiments, the first device may comprise a sensingunit. The sensing unit may comprise a color pixel array. The color pixelarray may be configured to output color information.

In some example embodiments, the first device may comprise a sensingunit. The sensing unit may comprise a depth pixel array and a colorpixel array. The depth pixel array may be configured to output depthinformation. The color pixel array may be configured to output colorinformation.

In some example embodiments, the first device may comprise a sensingunit. The sensing unit may comprise a pixel array. The pixel array maybe configured to output depth information, color information, or depthand color information.

In some example embodiments, a method for recognizing motion of anobject may comprise obtaining depth data of a first resolution withrespect to a scene; obtaining two-dimensional data of a secondresolution with respect to the scene; recognizing the object in thescene based on the depth data; tracking the object using a motiontracking region to provide tracking region data of the secondresolution; and/or analyzing the motion of the object based on thetracking region data. The second resolution may be higher than the firstresolution.

In some example embodiments, obtaining the depth data of the firstresolution may comprise using a depth pixel array of a sensing unit tooutput depth information.

In some example embodiments, obtaining the two-dimensional data of thesecond resolution may comprise using a color pixel array of a sensingunit to output color information.

In some example embodiments, obtaining the depth data of the firstresolution may comprise using a depth pixel array of a sensing unit tooutput depth information and/or obtaining the two-dimensional data ofthe second resolution may comprise using a color pixel array of thesensing unit to output color information.

In some example embodiments, obtaining the depth data of the firstresolution may comprise using a pixel array of a sensing unit to outputdepth information and/or obtaining the two-dimensional data of thesecond resolution may comprise using the pixel array of the sensing unitto output color information.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects and advantages will become more apparentand more readily appreciated from the following detailed description ofexample embodiments, taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a flowchart illustrating a method of recognizing a motion ofan object according to some example embodiments of the inventiveconcept;

FIG. 2 is a diagram illustrating an example of using a system accordingto some example embodiments of the inventive concept;

FIG. 3 is a block diagram illustrating a system according to someexample embodiments of the inventive concept;

FIG. 4 is a block diagram illustrating an example of an image capturingdevice in a motion recognizing device according to some exampleembodiments of the inventive concept;

FIG. 5 is a diagram illustrating an example of a sensing unit in theimage capturing device of FIG. 4;

FIG. 6 is a diagram illustrating an example of a pixel array in thesensing unit of FIG. 5;

FIG. 7 is a diagram illustrating a frame of an example depth dataobtained by an image capturing device;

FIG. 8 is a diagram illustrating a frame of an example two-dimensionaldata obtained by an image capturing device;

FIG. 9 is a diagram illustrating an example of a motion tracking regiondetermined according to some example embodiments of the inventiveconcept;

FIG. 10 is a diagram illustrating an example of a sensing unit in theimage capturing device of FIG. 4;

FIGS. 11A and 11B are diagrams illustrating example pixel arrays in thesensing unit of FIG. 10;

FIGS. 12A, 12B, 12C, and 12D are circuit diagrams illustrating someexample unit pixels in a pixel array;

FIG. 13 is a flowchart illustrating a method of recognizing a motion ofan object according to some example embodiments of the inventiveconcept;

FIG. 14 is a block diagram illustrating an example of a motion regiontracker in a motion recognizing device according to some exampleembodiments of the inventive concept;

FIGS. 15A, 15B, and 15C are diagrams illustrating example operations ofa synchronizer in the motion region tracker of FIG. 14;

FIG. 16 is a diagram for describing tracking region data provided by themotion region tracker of FIG. 14;

FIGS. 17A and 17B are diagrams for describing a method of upgrading amotion tracking region according to some example embodiments of theinventive concept;

FIG. 18 is a flowchart illustrating a method of recognizing a motion ofan object according to some example embodiments of the inventiveconcept;

FIG. 19 is a block diagram illustrating an example of a motion regiontracker in a motion recognizing device according to some exampleembodiments of the inventive concept;

FIG. 20 is a diagram for describing tracking region data provided by themotion region tracker of FIG. 19;

FIG. 21 is a flowchart illustrating a method of recognizing a motion ofan object according to some example embodiments of the inventiveconcept;

FIG. 22 is a block diagram illustrating an example of a motion regiontracker in a motion recognizing device according to some exampleembodiments of the inventive concept;

FIG. 23 is a diagram illustrating an example of a pixel array includedin a depth sensor;

FIG. 24 is a circuit diagram illustrating example time-of-flight (TOF)depth pixels in the pixel array of FIG. 23;

FIG. 25 is a timing diagram illustrating an operation of the TOF pixelsof FIG. 24;

FIG. 26 is a diagram illustrating an example combination for providingdepth data;

FIG. 27 is a diagram for describing a method of calculatingtwo-dimensional data based on raw data obtained using depth pixels;

FIGS. 28A and 28B are diagrams example combinations for providingtwo-dimensional data;

FIG. 29 illustrates a block diagram of a camera including athree-dimensional image sensor according to some example embodiments ofthe inventive concept;

FIG. 30 illustrates a block diagram of a computer system including amotion recognizing device according to some example embodiments of theinventive concept; and

FIG. 31 illustrates a block diagram of interface employable in thecomputing system of FIG. 30 according to some example embodiments of theinventive concept.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference tothe accompanying drawings. Embodiments, however, may be embodied in manydifferent forms and should not be construed as being limited to theembodiments set forth herein. Rather, these example embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope to those skilled in the art. In the drawings, thethicknesses of layers and regions may be exaggerated for clarity.

It will be understood that when an element is referred to as being “on,”“connected to,” “electrically connected to,” or “coupled to” to anothercomponent, it may be directly on, connected to, electrically connectedto, or coupled to the other component or intervening components may bepresent. In contrast, when a component is referred to as being “directlyon,” “directly connected to,” “directly electrically connected to,” or“directly coupled to” another component, there are no interveningcomponents present. As used herein, the term “and/or” includes any andall combinations of one or more of the associated listed items.

It will be understood that although the terms first, second, third,etc., may be used herein to describe various elements, components,regions, layers, and/or sections, these elements, components, regions,layers, and/or sections should not be limited by these terms. Theseterms are only used to distinguish one element, component, region,layer, and/or section from another element, component, region, layer,and/or section. For example, a first element, component, region, layer,and/or section could be termed a second element, component, region,layer, and/or section without departing from the teachings of exampleembodiments.

Spatially relative terms, such as “beneath,” “below,” “lower,” “above,”“upper,” and the like may be used herein for ease of description todescribe the relationship of one component and/or feature to anothercomponent and/or feature, or other component(s) and/or feature(s), asillustrated in the drawings. It will be understood that the spatiallyrelative terms are intended to encompass different orientations of thedevice in use or operation in addition to the orientation depicted inthe figures.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not intended to be limiting of exampleembodiments. As used herein, the singular forms “a,” “an,” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises,” “comprising,” “includes,” and/or “including,” when used inthis specification, specify the presence of stated features, integers,steps, operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

Example embodiments may be described herein with reference tocross-sectional illustrations that are schematic illustrations ofidealized example embodiments (and intermediate structures). As such,variations from the shapes of the illustrations as a result, forexample, of manufacturing techniques and/or tolerances, are to beexpected. Thus, example embodiments should not be construed as limitedto the particular shapes of regions illustrated herein but are toinclude deviations in shapes that result, for example, frommanufacturing. For example, an implanted region illustrated as arectangle will typically have rounded or curved features and/or agradient of implant concentration at its edges rather than a binarychange from implanted to non-implanted region. Likewise, a buried regionformed by implantation may result in some implantation in the regionbetween the buried region and the surface through which the implantationtakes place. Thus, the regions illustrated in the figures are schematicin nature, their shapes are not intended to illustrate the actual shapeof a region of a device, and their shapes are not intended to limit thescope of the example embodiments.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which example embodiments belong. Itwill be further understood that terms, such as those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andshould not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Reference will now be made to example embodiments, which are illustratedin the accompanying drawings, wherein like reference numerals may referto like components throughout.

FIG. 1 is a flowchart illustrating a method of recognizing a motion ofan object according to some example embodiments of the inventiveconcept.

Referring to FIG. 1, depth data ZDATA of a first resolution RES1 andtwo-dimensional data CDATA of a second resolution RES2 are obtained withrespect to a scene or a sequence of scenes using an image capturingdevice (S200). The second resolution RES2 is higher than the firstresolution RES1. For example, the first resolution RES1 may be thequarter video graphic array (QVGA) resolution of 320*240 pixels or alower resolution than the QVGA resolution. The second resolution RES2may be the high density (HD) resolution of 1280*720 pixels or a higherresolution than the HD resolution.

The depth data ZDATA represent the depth information or the distanceinformation of the scene captured by the image capturing device, and thetwo-dimensional data CDATA represent the image information of the scene.The depth data ZDATA may be provided in a form of a depth map in whichthe distance information of the respective portions of the scene isrepresented by unit of pixel. The two-dimensional data CDATA mayrepresent the color image or the black and white image of the scene.

The depth data ZDATA and the two-dimensional data CDATA may be providedperiodically by unit of frame. A first frame period corresponding to atime interval between the frames of the depth data ZDATA may be equal toor different from a second frame period corresponding to a time intervalbetween the frames of the two-dimensional data CDATA. For example, theframe rate of the depth data ZDATA may be 15 through 30 frames persecond (fps), and the frame rate of the two-dimensional data CDATA maybe 30 through 60 fps.

The scene may include a fixed background and a target object moving inthe background. A motion tracking region MTR is determined byrecognizing the target object in the scene based on the depth data ZDATA(S400). The motion tracking region MTR corresponds to a portion of theframe and the portion includes the target object. The recognition of thetarget object may be performed based on the depth data corresponding toa few frames that are sequentially captured.

The target object may be an independent thing such as a ball, a humanbody, etc. or a portion of the independent thing such as a hand, an arm,a head, etc. The recognition of the target object may be performed onvarious criteria according to the kind of the captured scene. Forexample, when the scene includes a plurality of moving objects, at leastone target object may be determined based on the kind, the distance, thespeed of the objects, etc. In some example embodiments, a simplethree-dimensional recognition algorithm may be used to determine thetarget object based on the depth data ZDATA. The motion tracking regionMTR may be determined based on such recognition of the target object. Ingeneral, the target object corresponds to a portion of the scene, andthus the motion tracking region MTR corresponds to a portion of theentire frame.

After the motion tracking region MTR is determined, tracking region dataTRDATA of the second resolution RES2 corresponding to the motiontracking region MTR are obtained periodically (S600). Some exampleembodiments of providing the tracking region data TRDATA will be furtherdescribed below. The amount of the tracking region data is less than thedata amount of the frame because the motion tracking region correspondsto a portion of the frame. As the size of the motion tracking region MTRto the size of the frame is decreased, the amount of the tracking regiondata TRDATA may be reduced significantly.

A motion of the target object is analyzed based on the tracking regiondata TRDATA (S800). The motion analysis may be performed by variousmethods known to those skilled in the art. However, compared with theconventional methods of analyzing the motion based on the datacorresponding to the entire frame, the motion analysis according to someexample embodiments may be based on the tracking region data TRDATA ofthe reduced amount and relatively the higher resolution, that is, thesecond resolution RES2.

As such, according to the method of recognizing the motion of someexample embodiments, the target object may be discerned using the depthdata ZDATA of the lower first resolution RES1, then the kind, the shape,and/or the motion of the target object may be analyzed using thetracking region data TRDATA of the higher second resolution RES2, andthus the motion of the target object may be better recognized. Inaddition, the data transfer and calculation amount for the motionrecognition may be reduced because the motion of the target object maybe analyzed based on the tracking region data TRDATA of the motiontracking region MTR corresponding to a portion of an entire frame.

FIG. 2 is a diagram illustrating an example of using a system accordingto some example embodiments of the inventive concept, and FIG. 3 is ablock diagram illustrating a system according to some exampleembodiments of the inventive concept.

As illustrated in FIG. 2, a system 10 may be a user interface systemthat operates by recognizing a motion of a user 15. The above-mentionedtarget object may include a body of the user 15 or a portion of the bodysuch as a hand, an arm, a head, a torso, etc. of the user 15.

Referring to FIGS. 2 and 3, the system 10 may include a motionrecognizing device 20 and a control device 30. The components for theuser interface such as a display device (DISP) 41, a speaker (SPK) 43, amicrophone (MIC) 45, etc., may be distinct from and coupled to thecontrol device 30 or may be formed as integral portions of the controldevice 30. The control device 30 may be an arbitrary device including amemory and a processor. For example, the control device 30 may be anotebook computer, a laptop computer, a television set, a game console,etc.

The motion recognizing device 20 may include an image capturing device(SEN) 100, a motion region tracker 300, and a motion analyzer (MANAL)500. The image capturing device 100 may be implemented with an imagecapturing device distinct from the other components.

At least a portion of the motion region tracker 300 and the motionanalyzer 500 may be implemented with software. In this case, thesoftware may be implemented with program codes that may be executed by ageneral purpose processor or an application processor. The program codesmay be implemented in a form of sub-routine that may be called andexecuted by a main program. According to some example embodiments, themotion analyzer 500 may be included in the control device 30, or themotion region tracker 300 and the motion analyzer 500 may be included inthe control device 30.

As described with reference to FIG. 1, the image capturing device 100obtains and provides the depth data ZDATA of the first resolution RES1and the two-dimensional data CDATA of the second resolution RES2periodically with respect to the scene the second resolution RES1 ishigher than the first resolution RES1 (S200). The motion region tracker300 determines the motion tracking region MTR by recognizing the targetobject in the scene corresponding to the frame based on the depth dataZDATA (S400), such that the motion tracking region MTR corresponds to aportion of the frame and the portion includes the target object. Inaddition, the motion region tracker 300 periodically obtains andprovides the tracking region data TRDATA of the second resolution RES2corresponding to the motion tracking region MTR (S600). The motionanalyzer 500 analyzes the motion of the target object based on thetracking region data TRDATA of the second resolution RES2 (S800).

Some example embodiments of the image capturing device 100 and themotion region tracker are further described with reference to FIGS. 4through 28B. The motion analyzer 500 may be implemented variously asknown to those skilled in the art, but it performs the motionrecognition not based on the data corresponding to the entire frame butbased on the tracking region data TRDATA corresponding to the motiontracking region MTR.

The control device 30 generates an event corresponding to the motion ofthe target object based on an analysis result of the motion analyzer500. The event may be transferred to the user through the display device41 and/or the speaker 43. For example, a display image corresponding tothe analyzed motion may be provided to the user through the displaydevice 41 and/or the sound corresponding to the analyzed motion may beprovided to the user through the speaker 43.

FIG. 4 is a block diagram illustrating an example of an image capturingdevice in a motion recognizing device according to some exampleembodiments of the inventive concept.

Referring to FIG. 4, the image capturing device 100 may include a lightsource (LS) 110, a sensing unit 130, and a control unit 150. The lightsource 110 generates a modulated transmission light TL to illuminate anobject with the modulated transmission light TL. The control unit 150generates control signals SYNC and CTRL to control operations of thelight source 110 and the sensing unit 130. The sensing unit 130 mayinclude depth pixels that convert reception light RL to electricalsignals. In addition, the sensing unit 130 may include color pixels thatconvert visible light VL to electrical signals.

The light source 110 may emit the modulated transmission light TL havinga given, desired, or predetermined wavelength. For example, the lightsource 110 may emit infrared light or near-infrared light. Thetransmission light TL generated by the light source 110 may be focusedon the object 60 by a lens 51. Modulated transmission light TL mayreflect from the object 60 to lens 53 as reception light RL and visiblelight VL.

The light source 110 may be controlled by the control signal SYNC tooutput the modulated transmission light TL such that the intensity ofthe modulated transmission light TL periodically changes. For example,the light source 110 may be implemented with a light emitting diode(LED), a laser diode, or the like.

The control signal SYNC from the control unit 150 may include a resetsignal RS and a transfer control signal TG, as will be described withreference to FIGS. 12A through 12D, and demodulation signals TG1 throughTG4 as will be described with reference to FIGS. 24 and 25. The controlsignal SYNC provided to the light source 110 may include a signal tosynchronize the modulated transmission light TL and the demodulationsignals TG1 through TG4.

The sensing unit 130 may include a pixel array PX in which depth pixelsand/or color pixels are arranged. Also the sensing unit 130 may includean analog-to-digital converting unit ADC and selection circuits ROW andCOL to select a particular pixel in the pixel array PX.

In some example embodiments, the image capturing device 100 may be athree-dimensional image sensor including the depth pixels for providingdistance information and the color pixels for providing imageinformation. In this case, the sensing unit 130 may include a pixelarray PX_CZ in which a plurality of depth pixels and a plurality ofcolor pixels are alternatively arranged as will be described withreference to FIG. 6.

In some example embodiments, the image capturing device 100 may includea depth sensor and a two-dimensional image sensor distinct from eachother. In this case, the sensing unit 130 may include a pixel array PX_Cin which a plurality of color pixels are arranged and a pixel array PX_Zin which a plurality of depth pixels are arranged as will be describedwith reference to FIGS. 11A and 11B.

In some example embodiments, the image capturing device 100 may includeonly a depth sensor. In this case, the sensing unit 130 may include apixel array PX_Z in which a plurality of depth pixels are arranged aswill be described with reference to FIG. 23.

In some example embodiments, the analog-to-digital converting unit ADCmay perform column analog-to-digital conversion that converts analogsignals in parallel using a plurality of analog-to-digital convertersrespectively coupled to a plurality of column lines, or may performsingle analog-to-digital conversion that converts the analog signals inseries using a single analog-to-digital converter.

In some example embodiments, the analog-to-digital converting unit ADCmay include a correlated double sampling (CDS) unit for extracting aneffective signal component (the valid voltage) based on the voltagessampled by the pixels.

In some example embodiments, the CDS unit may perform analog doublesampling (ADS) that extracts the effective signal component based on ananalog reset signal that represents a reset component and an analog datasignal that represents a signal component.

In some example embodiments, the CDS unit may perform digital doublesampling (DDS) that converts the analog reset signal and the analog datasignal into two digital signals to extract as the effective signalcomponent a difference between the two digital signals.

In some example embodiments, the CDS unit may perform dual correlateddouble sampling that performs both of analog double sampling and digitaldouble sampling.

FIG. 5 is a diagram illustrating an example of a sensing unit in theimage capturing device of FIG. 4. FIG. 5 illustrates an exampleembodiment of a sensing unit 130 a in a case where the image capturingdevice 100 of FIG. 4 is a three-dimensional image sensor.

Referring to FIG. 5, the sensing unit 130 a may include a pixel arrayPX_CZ where a plurality of color pixels and a plurality of depth pixelsare arranged, a color pixel select circuit CROW and CCOL, a depth pixelselect circuit ZROW and ZCOL, a color pixel converter CADC, and a depthpixel converter ZADC. The color pixel select circuit CROW and CCOL andthe color pixel converter CADC may provide color information RCDATA bycontrolling the color pixels included in the pixel array PX_CZ, and thedepth pixel select circuit ZROW and ZCOL and the depth pixel converterZADC may provide depth information RZDATA by controlling the depthpixels included in the pixel array PX_CZ.

The color information RCDATA and the depth information RZDATA from thesensing unit 130 a may be raw data and the above-describedtwo-dimensional data CDATA and the depth data ZDATA may be providedbased on the raw data. In the three-dimensional image sensor asillustrated in FIG. 5, components for controlling the color pixels andcomponents for controlling the depth pixels may independently operate toprovide the color information RCDATA and the depth information RZDATA ofthe captured image.

FIG. 6 is a diagram illustrating an example of a pixel array in thesensing unit of FIG. 5.

Referring to FIG. 6, the pixel array PX_CZ may include the color pixelsR, G, and B for providing the image information and the depth pixels Zfor providing the depth information. For example, the pixel pattern 101including the red pixel R, the green pixel G, the blue pixel B, and thedepth pixel Z may be repeatedly arranged in the pixel array PX_CZ.

Each of the color pixels R, G, and B may include a photo-detectionregion for collecting photo-electrons generated by the incident visiblelight, and the depth pixel Z may include a photo-detection region forcollecting photo-electrons generated by the reception light RL, that is,the incident infrared light or near-infrared light. For example, toenhance quantum efficiency, the depth pixel Z may include a photodiodeformed deeper than that of the color pixels R, G, and B since theinfrared light has a longer wavelength than that of the visible light.

Color filters may be formed over the color pixels R, G, and B andinfrared light pass filters may be formed over the depth pixels Z. Forexample, the red pixel R may be defined by the red filter, the greenpixel G may be defined by the green filter, the blue pixel B may bedefined by the blue filter, and the depth pixel Z may be defined by theinfrared light pass filter. In addition, infrared light cut filters maybe further formed over the color pixels R, G, and B.

FIG. 6 illustrates a non-limiting example of the pixel pattern 101, andthe pixel pattern 101 may be changed variously. For example, the arearatio of the one color pixel and the one depth pixel may be changedvariously and/or the number ratio of the color pixels and the depthpixels in the pixel array PX_CZ may be changed variously.

FIG. 7 is a diagram illustrating a frame of an example depth dataobtained by an image capturing device, FIG. 8 is a diagram illustratinga frame of an example two-dimensional data obtained by an imagecapturing device, and FIG. 9 is a diagram illustrating an example of amotion tracking region determined according to some example embodimentsof the inventive concept.

As illustrated in FIGS. 7 and 8, the image capturing device 100 providesthe frame of the depth data ZDATA has relatively a lower resolution,that is, the first resolution RES1 and the frame of the color data CDATAhas relatively a higher resolution, that is, the second resolution RES2.For example, the first resolution RES1 may be the quarter video graphicarray (QVGA) resolution of 320*240 pixels or a lower resolution than theQVGA resolution and the second resolution RES2 may be the high density(HD) resolution of 1280*720 pixels or a higher resolution than the HDresolution.

The motion region tracker 300 in FIG. 3 may recognize the target objectbased on the depth data ZDATA or the depth frame, and determines themotion tracking region MTR. The motion tracking region MTR correspondsto a portion of the frame and includes the target object in the scene.The scene may include a fixed background and the target object moving inthe background. For example, the target object may be a hand of a humanbeing as illustrated in FIG. 9, and the motion region tracker 300 mayrecognized the hand to determine the motion tracking region MTRincluding the hand.

The motion region tracker 300 may determine the motion tracking regionMTR based on the several frames of the depth data ZDATA. The targetobject may be an independent thing such as a ball, a human body, etc. ora portion of the independent thing such as a hand, an arm, a head, etc.The recognition of the target object may be performed on variouscriteria according to the kind of the captured scene. For example, whenthe scene includes a plurality of moving objects, at least one targetobject may be determined based on the kind, the distance, the speed ofthe objects, etc. In some example embodiments, a simplethree-dimensional recognition algorithm may be used to determine thetarget object based on the depth data ZDATA.

In some example embodiments, as illustrated in FIG. 9, the motiontracking region MTR may be determined by determining coordinates (x, y)of a center point of the motion tracking region MTR in the frame anddetermining a size (Lx, Ly) of the motion tracking region MTR in theframe. In some example embodiments, the motion tracking region MTR maybe determined by determining coordinates of four edge points of themotion tracking region MTR in the frame.

As shown in FIGS. 7, 8, and 9, the target object corresponds to aportion of the captured scene and thus the motion tracking regioncorresponds to a portion of the entire frame.

FIG. 10 is a diagram illustrating an example of a sensing unit in theimage capturing device of FIG. 4. FIG. 10 illustrates an exampleembodiment of a sensing unit 130 b in a case where the image capturingdevice 100 of FIG. 4 includes a depth sensor and a two-dimensional imagesensor distinct from each other.

Referring to FIG. 10, the sensing unit 130 b may include a pixel arrayPX_C where a plurality of color pixels are arranged and a pixel arrayPX_Z where a plurality of depth pixels are arranged. The visible lightVL for the image information and the reception light RL for the depthinformation may be separated by a beam splitter 55 and then illuminatedto the respective pixel arrays PX_C and PX_Z.

A color pixel select circuit CROW and CCOL, a depth pixel select circuitZROW and ZCOL, a color pixel converter CADC, and a depth pixel converterZADC may be disposed adjacent to the respective pixel arrays PX_C andPX_Z. The color pixel select circuit CROW and CCOL and the color pixelconverter CADC may provide the color information RCDATA by controllingthe color pixels included in the pixel array PX_C, and the depth pixelselect circuit ZROW and ZCOL and the depth pixel converter ZADC mayprovide the depth information RZDATA by controlling the depth pixelsincluded in the pixel array PX_Z. The color information RCDATA and thedepth information RZDATA from the sensing unit 130 b may be raw data andthe above-described two-dimensional data CDATA and the depth data ZDATAmay be provided based on the raw data.

As such, the sensing unit 130 b may include the depth sensor and thetwo-dimensional image sensor distinct from each other such that thecomponents for controlling the color pixels and the components forcontrolling the depth pixels may be implemented to respectively providethe color information RCDATA and the depth information RZDATA.

FIGS. 11A and 11B are diagrams illustrating example pixel arrays in thesensing unit of FIG. 10.

Referring to FIG. 11A, a first pixel array PX_C includes the colorpixels R, G, and B for providing the image information. For example, apixel pattern 102 including the green pixel G, the red pixel R, the bluepixel B, and the green pixel G may be repeatedly arranged in the firstpixel array PX_C. Each of the color pixels R, G, and B may include aphoto-detection region for collecting photo-electrons generated by theincident visible light. Color filters may be formed over the colorpixels R, G, and B. For example, the red pixel R may be defined by thered filter, the green pixel G may be defined by the green filter, and/orthe blue pixel B may be defined by the blue filter.

Referring to FIG. 11B, a second pixel array PX_Z includes the depthpixels Z for providing the depth information. For example, the identicaldepth pixels Z may be repeatedly arranged in the second pixel arrayPX_Z. Each of the depth pixels Z may include a photo-detection regionfor collecting photo-electrons generated by the reception light RL, thatis, the incident infrared light or near-infrared light. The infraredlight pass filter may be formed over each depth pixel Z.

FIGS. 12A, 12B, 12C, and 12D are circuit diagrams illustrating exampleunit pixels in a pixel array.

The unit pixels 200 a, 200 b, 200 c, and 200 d illustrated in FIGS. 12A,12B, 12C, and 12D may be a color pixel including a color photodiode or adepth pixel including a depth photodiode.

Referring to FIG. 12A, the unit pixel 200 a may include aphoto-sensitive element such as a photodiode PD, and a readout circuitincluding a transfer transistor TX, a reset transistor RX, a drivetransistor DX, and a selection transistor SX.

For example, the photodiode PD may include an n-type region in a p-typesubstrate such that the n-type region and the p-type substrate form ap-n conjunction diode. The photodiode PD receives the incident light andgenerates a photo-charge based on the incident light. In some exampleembodiments, the unit pixel 200 a may include a photo transistor, aphoto gate, a pinned photo diode, etc. instead of or in addition to thephotodiode PD.

The photo-charge generated in the photodiode PD may be transferred to afloating diffusion node FD through the transfer transistor TX, which isturned on in response to a transfer control signal TG. The drivetransistor DX functions as a source follower amplifier that amplifies asignal corresponding to the charge on the floating diffusion node FD.The selection transistor SX may transfer the amplified signal to acolumn line COL in response to a selection signal SEL. The floatingdiffusion node FD may be reset by the reset transistor RX. For example,the reset transistor RX may discharge the floating diffusion node FD inresponse to a reset signal RS for correlated double sampling (CDS).

FIG. 12A illustrates the unit pixel 200 a of the four-transistorconfiguration including the four transistors TX, RX, DX, and SX. Theconfiguration of the unit pixel may be variously changed as illustratedin FIGS. 12B, 12C, and 12D. Power is supplied via voltage supplyterminal VDD and ground.

Referring to FIG. 12B, the unit pixel 200 b may have thethree-transistor configuration including a photo-sensitive element suchas a photodiode PD, and a readout circuit including a reset transistorRX, a drive transistor DX, and a selection transistor SX. Compared withthe unit pixel 200 a of FIG. 12A, the transfer transistor TX is omittedin the unit pixel 200 b of FIG. 12B.

Referring to FIG. 12C, the unit pixel 200 c may have the five-transistorconfiguration including a photo-sensitive element such as a photodiodePD, and a readout circuit including a transfer transistor TX, a gatetransistor GX, a reset transistor RX, a drive transistor DX, and aselection transistor SX. The gate transistor GX may selectively applythe transfer control signal TG to the transfer transistor TX in responseto the selection signal SEL. Compared with the unit pixel 200 a of FIG.12A, the gate transistor GX is further included in the unit pixel 200 cof FIG. 12C.

Referring to FIG. 12D, the unit pixel 200 d may have the five-transistorconfiguration including a photo-sensitive element such as a photodiodePD, and a readout circuit including a photo transistor PX, a transfertransistor TX, a reset transistor RX, a drive transistor DX, and aselection transistor SX. The photo transistor PX may be turned on or offin response to a photo gate signal PG. The unit pixel 200 d may enabledwhen the photo transistor PX is turned on and disabled when the phototransistor PX is turned off. Compared with the unit pixel 200 a of FIG.12A, the photo transistor PX is further included in the unit pixel 200 dof FIG. 12D. In addition, the unit pixel may have six-transistorconfiguration further including the gate transistor GX of FIG. 12C (or abias transistor) in addition to the configuration of FIG. 12D.

FIG. 13 is a flowchart illustrating a method of recognizing a motion ofan object according to some example embodiment of the inventive concept.

Referring to FIG. 13, the image capturing device 100 as illustrated inFIG. 4 obtains and provides the depth data ZDATA corresponding to theframe with a first frame period PFRAME1 using the depth pixels Z of thefirst resolution RES1 (S212). In addition, the image capturing device100 obtains and provides the two-dimensional data CDATA corresponding tothe frame with a second frame period PFRAME2 using the color pixels R,G, and B of the second resolution RES2 (S214). In some exampleembodiments, the image capturing device 100 may include the one pixelarray PX_CZ in which the depth pixels Z and the color pixels R, G, and Bare alternatively arranged as illustrated in FIG. 6. In some exampleembodiments, the image capturing device 100 may include the two distinctpixel array PX_CZ in which the depth pixels Z and the color pixels R, G,and B are arranged respectively as illustrated in FIGS. 11A and 12B. Inother words, the image capturing device 100 may include athree-dimensional image sensor in some example embodiments, and/or thedepth sensor and the two-dimensional image sensor distinct from eachother in some example embodiments.

The motion region tracker 300 determines the motion tracking region MTRby recognizing the target object in the scene corresponding to the framebased on the depth data ZDATA (S400). The motion region tracker 300extracts region image data of the second resolution RES2 correspondingto the motion tracking region MTR from the two-dimensional data CDATA ofthe second resolution RES2 corresponding to the frame (S612) andprovides the region image data of the second resolution RES2 as thetracking region data TRDATA (S614). The motion analyzer 500 analyzes amotion of the target object based on the tracking region data TRDATA(S800) that are provided periodically.

FIG. 14 is a block diagram illustrating an example of a motion regiontracker in a motion recognizing device according to some exampleembodiments of the inventive concept.

Referring to FIG. 14, a motion region tracker 300 a may include a databuffer (BUFF) 310, a synchronizer (SYNC) 330, a tracking regiondeterminer (REGD) 350, and a data extractor (EXTR) 370.

The data buffer 310 may store the depth data ZDATA provided with thefirst frame period PFRAME1 and the two-dimensional data CDATA providedwith the second frame period PFRAME2 from the image capturing device100.

The synchronizer 330 may synchronize the depth data ZDATA and thetwo-dimensional data CDATA to be matched with each other, when the firstframe period PFRAME1 is different from the second frame period PFRAME2.The operation of the synchronizer 330 is further described withreference to FIGS. 15A, 15B and 15C. The synchronizer 330 may be omittedwhen the first frame period PFRAME1 is equal to the second frame periodPFRAME2.

The tracking region determiner 350 may determine the motion trackingregion MTR by recognizing the target object in the scene based on thedepth data ZDATA. The result of the determination may be provided to thedata extractor 370 as the coordinates (x, y) of the center point and thesize (Lx, Ly) of the motion tracking region MTR as described withreference to FIG. 9.

The data extractor 370 may extract the region image data of the secondresolution RES2 corresponding to the motion tracking region MTR from thetwo-dimensional data CDATA of the second resolution RES2 correspondingto the frame (S612) and provides the region image data of the secondresolution RES2 as the tracking region data TRDATA (S614).

As such, the target object may be discerned using the depth data ZDATAof the lower first resolution RES1, then the kind, the shape, and/or themotion of the target object may be analyzed using the tracking regiondata TRDATA of the higher second resolution RES2, and thus the motion ofthe target object may be better recognized.

FIGS. 15A, 15B, and 15C are diagrams illustrating example operations ofa synchronizer in the motion region tracker of FIG. 14.

As illustrated in FIGS. 15A, 15B, and 15C, the first frame periodPFRAME1 of the depth data ZDATA may be different from the second frameperiod PFRAME2 of the two-dimensional data CDATA. For example, the framerate (e.g., 60 fps) of the two-dimensional data CDATA may be three timesthe frame rate (e.g., 20 fps) of the depth data ZDATA.

In this case, the image capturing device 100 may sense and providefirst, second, and third two-dimensional frame data CF1, CF2, and CF3while sensing and providing first depth frame data DF1. In the same way,the image capturing device 100 may sense and provide fourth, fifth, andsixth two-dimensional frame data CF4, CF5, and CF6 while sensing andproviding second depth frame data DF2. The tracking region data TRDATAmay be provided with the first frame period PFRAME1 as illustrated inFIGS. 15A and 15B or with the second frame period PFRAME2 as illustratedin FIG. 15C, according to the operation of the synchronizer 330 in FIG.14.

Referring to FIG. 15A, the synchronizer 330 may synchronize the onedepth frame data and the one two-dimensional frame data, which areprovided simultaneously from the image capturing device 100 to providethe synchronized depth and two-dimensional frame data to the trackingregion determiner 350 and the data extractor 370. In other words, thesynchronizer 330 may match and provide the first depth frame data DF1and the third two-dimensional frame data CF3. In the same way, thesynchronizer 330 may match and provide the second depth frame data DF2and the sixth two-dimensional frame data CF6.

The tracking region determiner 350 may determine a first motion trackingregion MTR1 based on the first depth frame data DF1, and the dataextractor 370 may extract the portion corresponding to the first motiontracking region MTR1 from the third two-dimensional frame data CF3 toprovide first tracking region data TR1. In the same way, the trackingregion determiner 350 may determine a second motion tracking region MTR2based on the second depth frame data DF2, and the data extractor 370 mayextract the portion corresponding to the second motion tracking regionMTR2 from the sixth two-dimensional frame data CF6 to provide secondtracking region data TR2. As such, the first tracking region data TR1may be provided with the first frame period PFRAME1.

Referring to FIG. 15B, the synchronizer 330 may synchronize the onedepth frame data and the one two-dimensional frame data, which areprovided at different time points from the image capturing device 100 toprovide the synchronized depth and two-dimensional frame data to thetracking region determiner 350 and the data extractor 370. In otherwords, the synchronizer 330 may match and provide the first depth framedata DF1 and the second two-dimensional frame data CF2. In the same way,the synchronizer 330 may match and provide the second depth frame dataDF2 and the fifth two-dimensional frame data CF5. Since the first frameperiod PFRAME1 and the second frame period PFRAME2, the depth frame datamay be matched with the two-dimensional frame data corresponding to acenter of the sensing period for the corresponding depth frame data.

The tracking region determiner 350 may determine a first motion trackingregion MTR1 based on the first depth frame data DF1, and the dataextractor 370 may extract the portion corresponding to the first motiontracking region MTR1 from the second two-dimensional frame data CF2 toprovide first tracking region data TR1. In the same way, the trackingregion determiner 350 may determine a second motion tracking region MTR2based on the second depth frame data DF2, and the data extractor 370 mayextract the portion corresponding to the second motion tracking regionMTR2 from the fifth two-dimensional frame data CF5 to provide secondtracking region data TR2. As such, the first tracking region data TR1may be provided with the first frame period PFRAME1.

Referring to FIG. 15C, the synchronizer 330 may synchronize the onedepth frame data and the three two-dimensional frame data to provide thesynchronized depth and two-dimensional frame data to the tracking regiondeterminer 350 and the data extractor 370. In other words, thesynchronizer 330 may match and provide the first depth frame data DF1with the first, second, and third two-dimensional frame data CF1, CF2,and CF3. In the same way, the synchronizer 330 may match and provide thesecond depth frame data DF2 with the fourth, fifth, and sixthtwo-dimensional frame data CF4, CF5, and CF6.

The tracking region determiner 350 may determine a first motion trackingregion MTR1 based on the first depth frame data DF1, and the dataextractor 370 may extract the portions corresponding to the first motiontracking region MTR1 respectively from the first, second, and thirdtwo-dimensional frame data CF1, CF2, and CF3 to provide first, second,and third tracking region data TR1, TR2, and TR3. In the same way, thetracking region determiner 350 may determine a second motion trackingregion MTR2 based on the second depth frame data DF2, and the dataextractor 370 may extract the portions corresponding to the secondmotion tracking region MTR2 respectively from the fourth, fifth, andsixth two-dimensional frame data CF4, CF5, and CF6 to provide fourth,fifth, and sixth tracking region data TR4, TR5, and TR6. As such, thefirst tracking region data TR1 may be provided with the second frameperiod PFRAME2.

FIG. 16 is a diagram for describing tracking region data provided by themotion region tracker of FIG. 14.

FIG. 16 illustrates the two-dimensional image frame corresponding to thetwo-dimensional data CDATA and the tracking region data TRDATA extractedfrom the two-dimensional data CDATA. As described above, the portion ofthe two-dimensional data CDATA of the second resolution RES2, whichcorresponds to the motion tracking region MTR, may be provided as thetracking region data TRDATA. The data transfer and calculation amountfor the motion recognition may be reduced because the motion of thetarget object may be analyzed based on the tracking region data TRDATAof the motion tracking region MTR corresponding to a portion of anentire frame, and the motion recognition may be better performed becausethe tracking region data TRDATA has the higher second resolution RES2.

FIGS. 17A and 17B are diagrams for describing a method of upgrading amotion tracking region according to some example embodiments of theinventive concept.

FIGS. 17A and 17B illustrate the motion tracking regions MTR1 throughMTR4 and the corresponding tracking region data TRDATA1 through TRDATA4.The motion tracking region may be upgraded according to the motion ofthe target object, as illustrated in FIGS. 17A and 17B. In some exampleembodiments, the tracking region determiner 350 in FIG. 14 may detect achange of a position of the target object in the scene based on thedepth data ZDATA and change coordinates of a center point of the motiontracking region in the frame based on the change of the position of thetarget object in the scene. In some example embodiments, the trackingregion determiner 350 may detect a change of a distance to the targetobject based on the depth data ZDATA and change the size of the motiontracking region MTR based on the change of the distance to the targetobject.

FIG. 17A illustrates a case where the distance to the target object fromthe image capturing device 100 is increased, and FIG. 17B illustrates acase where the distance to the target object from the image capturingdevice 100 is decreased. When the distance to the target objectincreases as illustrated in FIG. 17A, the center coordinates of themotion tracking region may be changed from (x1, y1) to (x2, y2) and thesize of the motion tracking region may be decreased from (Lx1, Ly1) to(Lx2, Ly2). When the distance to the target object decreases asillustrated in FIG. 17B, the center coordinates of the motion trackingregion may be changed from (x3, y3) to (x4, y4) and the size of themotion tracking region may be increased from (Lx3, Ly3) to (Lx4, Ly4).

FIG. 18 is a flowchart illustrating a method of recognizing a motion ofan object according to some example embodiments of the inventiveconcept.

Referring to FIG. 18, the image capturing device 100 as illustrated inFIG. 4 obtains and provides the depth data ZDATA corresponding to theframe with the first frame period PFRAME1 using the depth pixels Z ofthe first resolution RES1 (S222). In addition, the image capturingdevice 100 obtains and provides the two-dimensional data CDATAcorresponding to the frame with the second frame period PFRAME2 usingthe color pixels R, G, and B of the second resolution RES2 (S224). Insome example embodiments, the image capturing device 100 may include theone pixel array PX_CZ in which the depth pixels Z and the color pixelsR, G, and B are alternatively arranged as illustrated in FIG. 6. In someexample embodiments, the image capturing device 100 may include the twodistinct pixel arrays PX_Z and PX_C in which the depth pixels Z and thecolor pixels R, G, and B are arranged respectively as illustrated inFIGS. 11A and 12B. In other words, the image capturing device 100 mayinclude a three-dimensional image sensor in some example embodiments,and/or the depth sensor and the two-dimensional image sensor distinctfrom each other in some example embodiments.

The motion region tracker 300 determines the motion tracking region MTRby recognizing the target object in the scene corresponding to the framebased on the depth data ZDATA (S400). The motion region tracker 300extracts region depth data of the first resolution RES1 corresponding tothe motion tracking region MTR from the depth data ZDATA of the firstresolution RES1 corresponding to the frame (S622), and extracts regionimage data of the second resolution RES2 corresponding to the motiontracking region MTR from the two-dimensional data CDATA of the secondresolution RES2 corresponding to the frame (S624). The motion regiontracker 300 compensates for the region depth data of the firstresolution RES1 using the region image data of the second resolutionRES2 to generate region depth data of the second resolution RES2 (S626),and provides the region depth data of the second resolution RES2 as thetracking region data TRDATA (S628). The motion analyzer 500 analyzes themotion of the target object based on the tracking region data TRDATA(S800) that are provided periodically.

FIG. 19 is a block diagram illustrating an example of a motion regiontracker in a motion recognizing device according to some exampleembodiments of the inventive concept.

Referring to FIG. 19, a motion region tracker 300 b may include a databuffer (BUFF) 310, a synchronizer (SYNC) 330, a tracking regiondeterminer (REGD) 350, a data extraction unit 371, and an image enhancer(ENH) 381.

The data buffer 310 may store the depth data ZDATA provided with thefirst frame period PFRAME1 and the two-dimensional data CDATA providedwith the second frame period PFRAME2 from the image capturing device100.

The synchronizer 330 may synchronize the depth data ZDATA and thetwo-dimensional data CDATA to be matched with each other, when the firstframe period PFRAME1 is different from the second frame period PFRAME2.The operation of the synchronizer 330 is the same as described withreference to FIGS. 15A, 15B, and 15C. The synchronizer 330 may beomitted when the first frame period PFRAME1 is equal to the second frameperiod PFRAME2.

The tracking region determiner 350 may determine the motion trackingregion MTR by recognizing the target object in the scene based on thedepth data ZDATA (S400). The result of the determination may be providedto the data extractor 370 as the coordinates (x, y) of the center pointand the size (Lx, Ly) of the motion tracking region MTR as describedwith reference to FIG. 9.

The data extraction unit 371 may include a first data extractor (EXTR1)372 and a second data extractor (EXTR2) 373. The first data extractor372 may extract region depth data of the first resolution RES1corresponding to the motion tracking region MTR from the depth dataZDATA of the first resolution RES1 corresponding to the frame (S622).The second data extractor 373 may extract region image data of thesecond resolution RES2 corresponding to the motion tracking region MTRfrom the two-dimensional data CDATA of the second resolution RES2corresponding to the frame (S624).

The image enhancer 381 may compensate for the region depth data of thefirst resolution RES1 using the region image data of the secondresolution RES2 to generate region depth data of the second resolutionRES2 (S626), and provide the region depth data of the second resolutionRES2 as the tracking region data TRDATA (S628). For example, the imageenhancer 381 may extract information of edges or textures from theregion image data of the second resolution RES2, and compensate for theregion depth data of the first resolution RES1 using the edge and/ortextures information to generate the region depth data of the secondresolution RES2.

As such, the target object may be discerned using the depth data ZDATAof the lower first resolution RES1, then the kind, the shape and/or themotion of the target object may be analyzed using the tracking regiondata TRDATA of the higher second resolution RES2, and thus the motion ofthe target object may be better recognized. Such a method of enhancingthe resolution of the depth data using the two-dimension data of thehigher resolution, with respect to only a portion of an entire frame,may be referred to as “local super-resolution”.

FIG. 20 is a diagram for describing tracking region data provided by themotion region tracker of FIG. 19.

FIG. 20 illustrates the depth frame corresponding to the depth dataZDATA′ and the tracking region data TRDATA′ extracted from the depthdata ZDATA′. As described above, the portion of the depth data ZDATA′ ofthe first resolution RES1, which corresponds to the motion trackingregion MTR, may be extracted. The extracted portion may be compensatedfor as described above and the compensated portion may be provided asthe tracking region data TRDATA′. As illustrated in FIG. 20, the edgesof the target object may be reinforced compared with the correspondingportion of the depth data ZDATA′. As such, improved analysis in additionto the reduction of data transfer and calculation amount may be achievedby enhancing the lower-resolution depth data corresponding to only aportion of an entire frame.

FIG. 21 is a flowchart illustrating a method of recognizing a motion ofan object according to some example embodiments of the inventiveconcept.

Referring to FIG. 21, the image capturing device 100 as illustrated inFIG. 4 obtains and provides periodically raw data RZDATA correspondingto the frame using time-of-flight (TOF) depth pixels (S232). In thiscase, the image capturing device 100 may include a single pixel arrayPX_Z in which the depth pixels Z are arranged as illustrated in FIG. 23.In other words, the image capturing device 100 may correspond to a depthsensor.

The motion region tracker 300 provides the depth data ZDATA of the firstresolution RES1 by combining every M bits of the raw data RZDATA (S234),where M is a positive integer equal to or greater than two, as will bedescribed with reference to FIG. 26, and provides the two-dimensionaldata CDATA of the second resolution RES2 by combining every N bits ofthe raw data RZDATA (S236), where N is a positive integer equal to orsmaller than M, as will be described with reference to FIGS. 28A and28B. In this case, the two-dimensional data CDATA may be black and whiteimage data BWDATA.

The motion region tracker 300 determines the motion tracking region MTRby recognizing the target object in the scene corresponding to the framebased on the depth data ZDATA (S400). The motion region tracker 300extracts region depth data of the first resolution RES1 corresponding tothe motion tracking region MTR from the depth data ZDATA of the firstresolution RES1 corresponding to the frame (S632), and extracts regionimage data of the second resolution RES2 corresponding to the motiontracking region MTR from the two-dimensional data CDATA of the secondresolution RES2 corresponding to the frame (S634). The motion regiontracker 300 compensates for the region depth data of the firstresolution RES1 using the region image data of the second resolutionRES2 to generate region depth data of the second resolution RES2 (S636),and provides the region depth data of the second resolution RES2 as thetracking region data TRDATA (S638). The motion analyzer 500 analyzes themotion of the target object based on the tracking region data TRDATA(S800) that are provided periodically.

FIG. 22 is a block diagram illustrating an example of a motion regiontracker in a motion recognizing device according to some exampleembodiments of the inventive concept.

Referring to FIG. 22, a motion region tracker 300 c may include a firstcalculator (CAL1) 305, a second calculator (CAL2) 306, a data buffer(BUFF) 315, a tracking region determiner (REGD) 355, a data extractionunit 375, and an image enhancer (ENH) 385.

The first calculator 305 may provide the depth data ZDATA of the firstresolution RES1 by combining every M bits of the raw data RZDATA (S234)that are provided periodically from the image capturing device 100,where M is a positive integer equal to or greater than two. The secondcalculator 306 may provide the two-dimensional data BWDATA of the secondresolution RES2 by combining every N bits of the raw data RZDATA (S236),where N is a positive integer equal to or smaller than M. The databuffer 315 may store the depth data ZDATA and the two-dimensional dataBWDATA provided periodically from the image capturing device 100. Thetracking region determiner 355 may determine the motion tracking regionMTR by recognizing the target object in the scene based on the depthdata ZDATA (S400). The result of the determination may be provided tothe data extraction unit 375 as the coordinates (x, y) of the centerpoint and the size (Lx, Ly) of the motion tracking region MTR asdescribed with reference to FIG. 9.

The data extraction unit 375 may include a first data extractor (EXTR1)376 and a second data extractor (EXTR2) 377. The first data extractor376 may extract region depth data of the first resolution RES1corresponding to the motion tracking region MTR from the depth dataZDATA of the first resolution RES1 corresponding to the frame (S632).The second data extractor 377 may extract region image data of thesecond resolution RES2 corresponding to the motion tracking region MTRfrom the two-dimensional data BWDATA of the second resolution RES2corresponding to the frame (S634).

The image enhancer 385 may compensate for the region depth data of thefirst resolution RES1 using the region image data of the secondresolution RES2 to generate region depth data of the second resolutionRES2 (S636), and provide the region depth data of the second resolutionRES2 as the tracking region data TRDATA (S638). For example, the imageenhancer 385 may extract information of edges or textures from theregion image data of the second resolution RES2, and compensate for theregion depth data of the first resolution RES1 using the edge and/ortextures information to generate the region depth data of the secondresolution RES2.

As such, the target object may be discerned using the depth data ZDATAof the lower first resolution RES1, then the kind, the shape, and/or themotion of the target object may be analyzed using the tracking regiondata TRDATA of the higher second resolution RES2, and thus the motion ofthe target object may be better recognized.

Using only a depth sensor without a two-dimensional or three-dimensionalimage sensor, the depth data ZDATA of the lower first resolution RES1and the two-dimensional data BWDATA of the higher second resolution RES2may be obtained, and the above-mentioned local super-resolution may beperformed such that the lower-resolution depth data may be enhancedusing the two-dimension data of the higher resolution, with respect toonly a portion of an entire frame.

FIG. 23 is a diagram illustrating an example of a pixel array includedin a depth sensor.

Referring to FIG. 23, the pixel array PX_Z includes a plurality of depthpixels Z1, Z2, Z3, and Z4. The depth pixels Z1, Z2, Z3, and Z4 may bethe time-of-flight (TOF) depth pixels that operate in response to aplurality of demodulation signals having different phases from eachother. For example, the depth pixel Z1 may operate in response to thedemodulation signal having phase difference of 0 degrees with respect tothe transmission light TL radiated from the image capturing device 100of FIG. 4. In other words, the depth pixel Z1 may operate in response tothe demodulation signal having the same phase with the transmissionlight TL. The depth pixel Z2 may operate in response to the demodulationsignal having phase difference of 90 degrees with respect to thetransmission light TL, the depth pixel Z3 may operate in response to thedemodulation signal having phase difference of 180 degrees with respectto the transmission light TL, and the depth pixel Z4 may operate inresponse to the demodulation signal having phase difference of 270degrees with respect to the transmission light TL. For example, thepixel pattern 103 including the depth pixels Z1, Z2, Z3, and Z4, whichoperate respectively in response to the demodulation signals of thedifferent phases, may be repeatedly arranged in the pixel array PX_Z.

FIG. 24 is a circuit diagram illustrating example time-of-flight (TOF)depth pixels in the pixel array of FIG. 23. FIG. 24 illustrates the onepixel pattern 103 in the pixel array PX_Z in FIG. 23.

Compared with the unit pixels of a single-tap structure in FIGS. 12A,12B, 12C, and 12D, first through fourth pixels Z1, Z2, Z3, and Z4 inFIG. 24 have a two-tap structure for measuring the distance according tothe TOF scheme.

Referring to FIG. 24, the first and third pixels Z1 and Z3 may share aphoto-sensitive element such as a photodiode PD. The first pixel Z1 mayinclude a first readout circuit including a first transfer transistorTX1, a first reset transistor RX1, a first drive transistor DX1, and afirst selection transistor SX1. The third pixel Z3 may include a thirdreadout circuit including a third transfer transistor TX3, a third resettransistor RX3, a third drive transistor DX3, and a third selectiontransistor SX3. In the same way, the second and fourth pixels Z2 and Z4may share a photo-sensitive element such as a photodiode PD. The secondpixel Z2 may include a second readout circuit including a secondtransfer transistor TX2, a second reset transistor RX2, a second drivetransistor DX2, and a second selection transistor SX2. The fourth pixelZ4 may include a fourth readout circuit including a fourth transfertransistor TX4, a fourth reset transistor RX4, a fourth drive transistorDX4, and a fourth selection transistor SX4.

For example, the photodiode PD may include an n-type region in a p-typesubstrate such that the n-type region and the p-type substrate form ap-n conjunction diode. The photodiode PD receives the incident light andgenerates a photo-charge based on the incident light. In some exampleembodiments, the unit pixel 200 e illustrated in FIG. 24 may include aphoto transistor, a photo gate, a pinned photo diode, etc. instead of orin addition to the photodiode PD.

The photo-charge generated in the photodiodes PD may be transferred tofloating diffusion nodes FD1, FD2, FD3, and FD4 through the transfertransistors TX1, TX2, TX3, and TX4, respectively. The transfer controlsignals TG1, TG2, TG3, and TG4 may be the above-described demodulationsignals having the phase difference of 0, 90, 180, and 270 degrees,respectively, with respect to the transmission light TL. As such, thephoto charge generated in the photodiodes PD may be divided in responseto the demodulation signals TG1, TG2, TG3, and TG4 to determine theroundtrip TOF of the light and the distance to the object may becalculated based on the roundtrip TOF.

The drive transistors DX1, DX2, DX3, and DX4 function as source followeramplifiers that amplify signals corresponding to the respective chargeson the floating diffusion nodes FD1, FD2, FD3, and FD4. The selectiontransistors SX1, SX2, SX3, and SX4 may transfer the amplified signals tothe column lines COL1 and COL2 in response to the selection signalsSEL1, SEL2, SEL3, and SEL4, respectively. The floating diffusion nodesFD1, FD2, FD3, and FD4 may be reset by the reset transistors RX1, RX2,RX3, and RX4, respectively. For example, the reset transistors RX1, RX2,RX3, and RX4 may discharge the floating diffusion nodes FD1, FD2, FD3,and FD4 in response to reset signals RS1, RS2, RS3, and RS4,respectively, for correlated double sampling (CDS).

FIG. 24 illustrates the non-limiting example of the depth pixels of thetwo-tap configuration, and the depth pixels may have variousconfigurations such as a single-tap configuration, four-tapconfiguration, etc. The timing of the control signals may be determinedproperly depending on the configuration of the depth pixels.

FIG. 25 is a timing diagram illustrating an operation of the TOF pixelsof FIG. 24.

Referring to FIG. 25, an object is illuminated with a modulatedtransmission light TL during an integration time interval TINT. Asdescribed with reference to FIG. 4, the image capturing device 100 mayinclude a light source 110 or a light-emitting device to generate themodulated transmission light TL having periodically varying intensity.For example, the image capturing device 100 may repeat the transmissionand non-transmission of the modulated transmission light TL by turningon or off the light-emitting device at a frequency ranging from about 10MHz to about 200 MHz. Even though FIG. 25 illustrates the modulatedtransmission light TL of a pulse train, an arbitrary periodic lightsignal such as a sinusoidal signal may be used as the modulatedtransmission light TL and the demodulation signals TG1, TG2, TG3, andTG4.

The modulated transmission light TL is reflected by the object andreturned to the image capturing device 100 as a reception light RL. Thereception light RL is delayed by a time-of-flight (TOF) with respect tothe modulated transmission light TL. The photo-charge is generated inthe photo-detection region of the depth pixel by the reception light RL.

The demodulation signals TG1, TG2, TG3, and TG4 may have a given,desired, or predetermined phases with respect to the modulatedtransmission light TL. If the photo-charges Q1, Q2, Q3, and Q4integrated during the activation interval of the demodulation signalsTG1, TG2, TG3, and TG4 are obtained, the TOF may be calculated based onthe photo-charges Q1, Q2, Q3, and Q4.

When the distance from the photo-sensing device to the object is ‘D’ anda light velocity is ‘c’, the distance may be calculated using therelation D=(TOF*c)/2. Even though FIG. 25 illustrates the fourdemodulation signals TG1, TG2, TG3, and TG4 having the different phases,a different combination of the demodulation signals may be used toobtain the TOF. For example, the image capturing device 100 may use onlythe first demodulation signal TG1 having a phase equal to a phase of themodulated transmission light TL and a third demodulation signal TG3having a phase opposite to the phase of the modulated transmission lightTL. Even though not illustrated in FIG. 25, the photo-detection regionsPD and the floating diffusion regions FD may be initialized byactivating the reset signal RS, etc. before the integration timeinterval TINT.

During a readout time interval TRD, the data bits D1, D2, D3, and D4corresponding to the integrated photo-charges Q1, Q2, Q3, and Q4 areprovided through column lines COL1 and COL2. The data bits D1, D2, D3,and D4 may be output as the above-described raw data RZDATA and used incalculating the depth data ZDATA and/or the two-dimensional data CDATA.

FIG. 26 is a diagram illustrating an example combination for providingdepth data.

Referring to FIG. 26, one bit value of the depth data ZDATA may beprovided based on four bit values of the raw data RZDATA, where the fourbit values respectively correspond to the four demodulation signals TG1,TG2, TG3, and TG4 having the phase difference of 0, 90, 180, and 270degrees, respectively, with respect to the transmission light TL. Inother words, each combination CMB including the four depth pixels Z1,Z2, Z3, and Z4 may provide the one bit value of the depth data ZDATA.

As described above, the one bit value of the depth data ZDATA may beobtained by calculating the TOF based on the photo-charges Q1, Q2, Q3,and Q4 respectively corresponding to the demodulation signals TG1, TG2,TG3, and TG4 having the phase differences of 0, 90, 180, and 270degrees. In FIG. 26, the small solid rectangular points are illustratedto indicate the positions corresponding to the bits of the calculateddepth data ZDATA. When the depth pixels are arranged in a matrix formwith the resolution of 2X*2Y, the above-described first resolution RES1,i.e., the resolution of the depth data ZDATA may be X*Y.

FIG. 27 is a diagram for describing a method of calculatingtwo-dimensional data based on raw data obtained using depth pixels.

Referring to FIG. 27, one bit value of the two-dimensional data BWDATAmay be obtained by summing two bit values corresponding to the twodemodulation signals having the opposite phases among the four bitvalues D1, D2, D3, and D4 of the raw data RZDATA. One bit value D1+D3 ofthe two-dimensional data BWDATA may be obtained by summing two bitvalues D1 and D3 of the raw data RZDATA, where the two bit values D1 andD3 respectively correspond to the two demodulation signals GT1 and GT3having the phase differences of 0 and 180 degrees, among the four bitvalues D1, D2, D3, and D4 of the raw data RZDATA. In addition, anotherbit value D2+D4 of the two-dimensional data BWDATA may be obtained bysumming other two bit values D2 and D4 of the raw data RZDATA, where theother two bit values D2 and D4 respectively correspond to the twodemodulation signals GT2 and GT4 having the phase differences of 90 and270 degrees, among the four bit values D1, D2, D3, and D4 of the rawdata RZDATA. As such, by summing the two bit values of the raw dataRZDATA corresponding to the two demodulation signals having the oppositephases, the calculated two-dimensional data BWDATA may represent theblack and white image.

FIGS. 28A and 28B are diagrams example combinations for providingtwo-dimensional data.

As described with reference to FIG. 27, each bit value of thetwo-dimensional data BWDATA may be provided by summing the two bitvalues of the raw data RZDATA.

Referring to FIG. 28A, according to the combinations CMB1 and CMB3, theone bit value of the two-dimensional data BWDATA may be provided bysumming the two bit values from the two pixels Z1 and Z3 correspondingto the demodulation signals GT1 and GT3 having the phase differences of0 and 180 degrees with respect to the transmission light TL. Inaddition, according to the combinations CMB2 and CMB4, the one bit valueof the two-dimensional data BWDATA may be provided by summing the twobit values from the two pixels Z2 and Z4 corresponding to thedemodulation signals GT2 and GT4 having the phase differences of 90 and270 degrees with respect to the transmission light TL.

In FIG. 28A, the small solid circular points are illustrated to indicatethe positions corresponding to the bits of the calculatedtwo-dimensional data BWDATA. When the depth pixels are arranged in amatrix form with the resolution of 2X*2Y, the above-described secondresolution RES2, i.e., the resolution of the two-dimensional data BWDATAmay be 2X*(2Y−1).

Referring to FIG. 28B, according to the combinations CMB5, CMB6, CMB7,and CMB8, the one bit value of the two-dimensional data BWDATA may beprovided by summing the four bit values from the four pixels Z1, Z2, Z3,and Z4 corresponding to the demodulation signals GT1, GT2, GT3, and GT4having the phase differences of 0, 90, 180, and 270 degrees with respectto the transmission light TL.

In FIG. 28B, the small solid triangular points are illustrated toindicate the positions corresponding to the bits of the calculatedtwo-dimensional data BWDATA. When the depth pixels are arranged in amatrix form with the resolution of 2X*2Y, the above-described secondresolution RES2, i.e., the resolution of the two-dimensional data BWDATAmay be (2X−1)*(2Y−1).

The depth data ZDATA of the lower first resolution RES1 (e.g., X*Y) maybe provided by combining M bits (e.g., four bits) of the raw data RZDATAas described with reference to FIG. 27, and the two-dimensional dataBWDATA of the higher second resolution RES2 (e.g., 2X*(2Y−1) or(2X−1)*(2Y−1)) may be provided by combining N bits (e.g., two bits orfour bits) of the raw data RZDATA as described with reference to FIGS.28A and 28B.

As such, the depth data ZDATA and the two-dimensional data BWDATA may beobtained using only the depth sensor without the two-dimensional imagesensor, and the above-mentioned local super-resolution may be performedusing the obtained data ZDATA and BWDATA.

FIG. 29 illustrates a block diagram of a camera including athree-dimensional image sensor according to some example embodiments ofthe inventive concept.

Referring to FIG. 29, a camera 800 includes a photo-receiving lens 810,a three-dimensional image sensor 900, and an engine unit 840. Thethree-dimensional image sensor 900 may include a three-dimensional imagesensor chip 820 and a light source module 830. According to some exampleembodiments, the three-dimensional image sensor chip 820 and the lightsource module 830 may be implemented with separated devices, or at leasta portion of the light source module 830 may be included in thethree-dimensional image sensor chip 820. In some example embodiments,the photo-receiving lens 810 may be included in the three-dimensionalimage sensor chip 820.

The photo-receiving lens 810 may focus incident light on aphoto-receiving region (e.g., depth pixels and/or color pixels includedin a pixel array) of the three-dimensional image sensor chip 820. Thethree-dimensional image sensor chip 820 may generate data DATA1including depth information and/or color image information based on theincident light passing through the photo-receiving lens 810. Forexample, the data DATA1 generated by the three-dimensional image sensorchip 820 may include depth data generated using infrared light ornear-infrared light emitted from the light source module 830 and red,green, blue (RGB) data of a Bayer pattern generated using externalvisible light. The three-dimensional image sensor chip 820 may providethe data DATA1 to the engine unit 840 based on a clock signal CLK. Insome example embodiments, the three-dimensional image sensor chip 820may interface with the engine unit 840 via mobile industry processorinterface (MIPI®) and/or camera serial interface (CSI).

The engine unit 840 controls the three-dimensional image sensor 900. Theengine unit 840 may process the data DATA1 received from thethree-dimensional image sensor chip 820. To perform the above-describedmethod of recognizing the motion according to some example embodiments,the engine unit 840 may include motion region tracker 300 and/or themotion analyzer 500. The engine unit may perform data processing inaddition to the motion recognition. For example, the engine unit 840 maygenerate three-dimensional color data based on the data DATA1 receivedfrom the three-dimensional image sensor chip 820. In other examples, theengine unit 840 may generate luminance, chrominance (YUV) data includinga luminance component Y, a blue-luminance difference component U, and ared-luminance difference component V based on the RGB data included inthe data DATA1, or compressed data, such as Joint Photographic ExpertsGroup (JPEG) data. The engine unit 840 may be connected to ahost/application 850 and may provide data DATA2 to the host/application850 based on a master clock MCLK. Further, the engine unit 840 mayinterface with the host/application 850 via serial peripheral interface(SPI) and/or inter integrated circuit (I2C).

FIG. 30 illustrates a block diagram of a computer system including amotion recognizing device according to some example embodiments of theinventive concept.

Referring to FIG. 30, a computing system 1000 may include a processor1010, a memory device 1020, a storage device 1030, an input/outputdevice 1040, a power supply 1050, and a three-dimensional image sensor900. Although it is not illustrated in FIG. 30, the computing system1000 may further include ports that communicate with a video card, asound card, a memory card, a universal serial bus (USB) device, and/orother electronic devices.

The processor 1010 may perform various calculations or tasks. Accordingto some embodiments, the processor 1010 may be a microprocessor or acentral processing unit (CPU). The processor 1010 may communicate withthe memory device 1020, the storage device 1030, and the input/outputdevice 1040 via an address bus, a control bus, and/or a data bus. Insome example embodiments, the processor 1010 may be coupled to anextended bus, such as a peripheral component interconnection (PCI) bus.The memory device 1020 may store data for operating the computing system1000. For example, the memory device 1020 may be implemented with adynamic random access memory (DRAM) device, a mobile DRAM device, astatic random access memory (SRAM) device, a phase random access memory(PRAM) device, a ferroelectric random access memory (FRAM) device, aresistive random access memory (RRAM) device, and/or a magnetic randomaccess memory (MRAM) device. The storage device may include a solidstate drive (SSD), a hard disk drive (HDD), a compact-disc read-onlymemory (CD-ROM), etc. The input/output device 1040 may include an inputdevice (e.g., a keyboard, a keypad, a mouse, etc.) and an output device(e.g., a printer, a display device, etc.). The power supply 1050supplies operation voltages for the computing system 1000.

The three-dimensional image sensor 900 may communicate with theprocessor 1010 via the buses or other communication links. Thethree-dimensional image sensor 900 may be integrated with the processor1010 in one chip, or the three-dimensional image sensor 900 and theprocessor 1010 may be implemented as separate chips.

The three-dimensional image sensor 900 may be packaged in various forms,such as package on package (PoP), ball grid arrays (BGAs), chip scalepackages (CSPs), plastic leaded chip carrier (PLCC), plastic dualin-line package (PDIP), die in waffle pack, die in wafer form, chip onboard (COB), ceramic dual in-line package (CERDIP), plastic metric quadflat pack (MQFP), thin quad flat pack (TQFP), small outline integratedcircuit (SOIC), shrink small outline package (SSOP), thin small outlinepackage (TSOP), system in package (SIP), multi-chip package (MCP),wafer-level fabricated package (WFP), or wafer-level processed stackpackage (WSP).

The computing system 1000 may be any computing system using athree-dimensional image sensor. For example, the computing system 1000may include a digital camera, a mobile phone, a smart phone, a portablemultimedia player (PMP), a personal digital assistant (PDA), etc.

FIG. 31 illustrates a block diagram of an interface employable in thecomputing system of FIG. 30 according to some example embodiments of theinventive concept.

Referring to FIG. 31, a computing system 1100 may be implemented by adata processing device that uses or supports a mobile industry processorinterface (MIPI®) interface. The computing system 1100 may include anapplication processor 1110, a three-dimensional image sensor 1140, adisplay device 1150, etc. A CSI host 1112 of the application processor1110 may perform a serial communication with a CSI device 1141 of thethree-dimensional image sensor 1140 via a camera serial interface (CSI).In some example embodiments, the CSI host 1112 may include adeserializer (DES), and the CSI device 1141 may include a serializer(SER). A DSI host 1111 of the application processor 1110 may perform aserial communication with a DSI device 1151 of the display device 1150via a display serial interface (DSI).

In some example embodiments, the DSI host 1111 may include a serializer(SER), and the DSI device 1151 may include a deserializer (DES). Thecomputing system 1100 may further include a radio frequency (RF) chip1160 performing a communication with the application processor 1110 anda DigRF℠ slave 1162 providing communication with other devices. Aphysical layer (PHY) 1113 of the computing system 1100 and a physicallayer (PHY) 1161 of the RF chip 1160 may perform data communicationsbased on a MIPI® DigRF℠. The application processor 1110 may furtherinclude a DigRF℠ MASTER 1114 that controls the data communications ofthe PHY 1161.

The computing system 1100 may further include a global positioningsystem (GPS) 1120, a storage 1170, a MIC 1180, a DRAM device 1185, and aspeaker 1190. In addition, the computing system 1100 may performcommunications using an ultra-wideband (UWB) 1210, a wireless local areanetwork (WLAN) 1220, a worldwide interoperability for microwave access(WIMAX) 1230, etc. However, the structure and the interface of thecomputing system 1100 are not limited thereto.

Some example embodiments of the inventive concept may be applied toarbitrary devices and/or systems that require rapid and/or improvedmotion recognition for the moving object. Particularly, some exampleembodiments of the inventive concept may be applied usefully to thedevices and/or systems requiring a user interface based on the motion ofthe user.

It should be understood that the exemplary embodiments described thereinshould be considered in a descriptive sense only and not for purposes oflimitation. Descriptions of features or aspects within each embodimentshould typically be considered as available for other similar featuresor aspects in other embodiments.

What is claimed is:
 1. A method of recognizing motion of an object, themethod comprising: periodically obtaining depth data of a firstresolution and two-dimensional data of a second resolution with respectto a scene using an image capturing device, wherein the secondresolution is higher than the first resolution; determining a motiontracking region by recognizing a target object in the scene based on thedepth data, such that the motion tracking region corresponds to aportion of a frame and the portion includes the target object;periodically obtaining tracking region data of the second resolutioncorresponding to the motion tracking region; and analyzing motion of thetarget object based on the tracking region data, wherein theperiodically obtaining the tracking region data of the second resolutionincludes, extracting region depth data of the first resolutioncorresponding to the motion tracking region from the depth data of thefirst resolution corresponding to the frame; extracting region imagedata of the second resolution corresponding to the motion trackingregion from the two-dimensional data of the second resolutioncorresponding to the frame; compensating for the region depth data ofthe first resolution using the region image data of the secondresolution to generate region depth data of the second resolution; andproviding the region depth data of the second resolution as the trackingregion data.
 2. The method of claim 1, wherein periodically obtainingthe depth data and the two-dimensional data includes: periodicallyproviding the depth data corresponding to the frame using time-of-flight(TOF) depth pixels, the TOF depth pixels operating in response to aplurality of demodulation signals having different phases from eachother.
 3. The method of claim 1, wherein determining the motion trackingregion within the scene includes: determining coordinates of a centerpoint of the motion tracking region within the scene in the frame; anddetermining a size of the motion tracking region within the scene in theframe.
 4. The method of claim 1, wherein periodically obtaining thedepth data and the two-dimensional data includes: providing the depthdata corresponding to the frame with a first frame period using depthpixels of the first resolution; and providing the two-dimensional datacorresponding to the frame with a second frame period using color pixelsof the second resolution.
 5. The method of claim 4, further comprising:synchronizing the depth data and the two-dimensional data to be matchedwith each other, when the first frame period is different from thesecond frame period.
 6. A method of recognizing motion of an object, themethod comprising: arranging depth pixels and color pixels in a commonpixel array; periodically obtaining depth data of a first resolution andtwo-dimensional data of a second resolution with respect to a sceneusing the depth pixels and the color pixels, wherein the secondresolution is higher than the first resolution; determining a motiontracking region by recognizing a target object in the scene based on thedepth data, such that the motion tracking region corresponds to aportion of a frame and the portion includes the target object;periodically obtaining tracking region data of the second resolutioncorresponding to the motion tracking region; and analyzing motion of thetarget object based on the tracking region data.
 7. The method of claim6, wherein the depth pixels correspond to time-of-flight (TOF) depthpixels, the TOF depth pixels operating in response to a plurality ofdemodulation signals having different phases from each other.
 8. Themethod of claim 7, wherein one bit value of the depth data is providedbased on M bit values of the raw data and one bit value of thetwo-dimensional data is provided based on N bit values of the raw data,where M is a natural number and N is a natural number smaller than N. 9.The method of claim 7, wherein periodically obtaining the depth data andthe two-dimensional data includes: periodically providing raw datacorresponding to the frame using the TOF depth pixels; and calculatingthe depth data of the first resolution and the two-dimensional data ofthe second resolution based on the raw data.
 10. The method of claim 9,wherein calculating the depth data of the first resolution and thetwo-dimensional data of the second resolution includes: providing thedepth data of the first resolution by combining every M bits of the rawdata, where M is a positive integer equal to or greater than two; andproviding the two-dimensional data of the second resolution by combiningevery N bits of the raw data, where N is a positive integer equal to orsmaller than M.
 11. The method of claim 9, wherein, the depth pixels andcolor pixels are included in an image capturing device, and thedemodulation signals have phase differences of 0, 90, 180 and 270degrees, respectively, with respect to transmission light radiated fromthe image capturing device.
 12. The method of claim 11, whereinproviding the depth data of the first resolution includes: providing onebit value of the depth data based on four bit values of the raw data,the four bit values respectively corresponding to the four demodulationsignals having the phase differences of 0, 90, 180 and 270 degrees,respectively.
 13. The method of claim 11, wherein providing thetwo-dimensional data of the second resolution includes: providing onebit value of the two-dimensional data by summing two bit values of theraw data, the two bit values respectively corresponding to the twodemodulation signals having the phase differences of 0 and 180 degrees;and providing another bit value of the two-dimensional data by summingother two bit values of the raw data, the other two bit valuesrespectively corresponding to the two demodulation signals having thephase differences of 90 and 270 degrees.
 14. The method of claim 6,wherein periodically obtaining the tracking region data of the secondresolution includes: extracting region image data of the secondresolution corresponding to the motion tracking region within the scenefrom the two-dimensional data of the second resolution corresponding tothe frame; and providing the region image data of the second resolutionas the tracking region data.
 15. The method of claim 6, whereinperiodically obtaining the tracking region data of the second resolutionincludes: extracting region depth data of the first resolutioncorresponding to the motion tracking region within the scene from thedepth data of the first resolution corresponding to the frame;extracting region image data of the second resolution corresponding tothe motion tracking region within the scene from the two-dimensionaldata of the second resolution corresponding to the frame; compensatingfor the region depth data of the first resolution using the region imagedata of the second resolution to generate region depth data of thesecond resolution; and providing the region depth data of the secondresolution as the tracking region data.
 16. An apparatus for recognizingmotion of an object, the apparatus comprising: an image capturing deviceconfigured to periodically provide depth data of a first resolution andtwo-dimensional data of a second resolution with respect to a scene,wherein the second resolution is higher than the first resolution; amotion region tracker configured to determine a motion tracking regionwithin the scene by recognizing a target object in the scene based onthe depth data, such that the motion tracking region within the scenecorresponds to a portion of a frame and the portion includes the targetobject, and configured to periodically provide tracking region data ofthe second resolution corresponding to the motion tracking region withinthe scene; and a motion analyzer configured to analyze motion of thetarget object based on the tracking region data, wherein the imagecapturing device includes: a pixel array in which time-of-flight (TOF)depth pixels of the first resolution and color pixels of the secondresolution are alternatively arranged, the TOF depth pixels operating inresponse to a plurality of demodulation signals having different phasesfrom each other to periodically provide raw data corresponding to theframe.
 17. The apparatus of claim 16, wherein the demodulation signalshave phase differences of 0, 90, 180, and 270 degrees, respectively,with respect to transmission light radiated from the image capturingdevice, and one bit value of the depth data is provided based on fourbit values of the raw data, the four bit values respectivelycorresponding to the four demodulation signals having the phasedifferences of 0, 90, 180, and 270 degrees, respectively.
 18. Theapparatus of claim 16, wherein the image capturing device furtherincludes: a color pixel select circuit and a color pixel converterconfigured to provide the two-dimensional data by controlling the colorpixels; and a depth pixel select circuit and a depth pixel converterconfigured to provide the depth pixel by controlling the TOF depthpixels.
 19. The apparatus of claim 16, wherein the motion region trackeris configured to: extract region image data of the second resolutioncorresponding to the motion tracking region within the scene from thetwo-dimensional data of the second resolution corresponding to theframe; and provide the region image data of the second resolution as thetracking region data.
 20. The apparatus of claim 16, wherein the motionregion tracker is configured to: extract region depth data of the firstresolution corresponding to the motion tracking region within the scenefrom the depth data of the first resolution corresponding to the frame;extract region image data of the second resolution corresponding to themotion tracking region within the scene from the two-dimensional data ofthe second resolution corresponding to the frame; compensate for theregion depth data of the first resolution using the region image data ofthe second resolution to generate region depth data of the secondresolution; and provide the region depth data of the second resolutionas the tracking region data.